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Abstract — We  describe  the  design  of  human-machine  conver¬ 
sation  experiments  that  support  the  evaluation  of  our  context- 
aware  approach  in  coalition  decision  making  at  or  near  the 
network  edge.  The  studies,  named  SHERLOCK  (for  Simple 
Human  Experiments  Regarding  Locally  Observed  Collective 
Knowledge),  involve  humans  participating  in  simple  intelligence, 
surveillance,  and  reconnaissance  (ISR)  tasks  in  physical  envi¬ 
ronments,  using  mobile  devices  in  situ,  or  online.  Experimental 
tasks  are  undertaken  by  multiple  participants  operating  as  a 
coalition,  with  collaboration  mediated  by  machine  agents.  We 
illustrate  the  SHERLOCK  using  two  specific  experiment  designs. 
In  each  one,  the  participants’  task  is  to  locate  a  number  of  target 
individuals  and  identify  their  specific  features.  Experiment  1  is 
a  crowdsourcing  “whodunit”  scenario  involving  intelligence  in 
synthetic,  natural,  and  hidden  situations.  All  human  input  is 
via  a  conversational  agent  which  mediates  information  sharing 
between  participants.  Experiment  2  is  an  ISR  asset  assignment 
scenario,  using  simulated  sensing  assets  selected  either  by  a 
human,  via  an  algorithm,  or  a  combination  of  the  human  and 
machine.  Participants’  success  in  both  experiments  depends  on 
their  ability  to  make  effective  use  of  the  conversational  system, 
including  being  able  to  provide  interpretable  information  to 
the  conversational  agent,  and  to  obtain  information  via  the 
conversational  interface  to  assist  them. 

I.  Introduction 

Over  the  past  four  years  we  have  been  researching  technolo¬ 
gies  to  support  human-machine  collaboration  in  the  context  of 
coalition  intelligence,  surveillance,  and  reconnaissance  (ISR) 
tasks  [1],  We  have  focussed  on  approaches  using  controlled 
natural  language  (CNL)  to  provide  representations  of  infor¬ 
mation  and  knowledge  that  are  human  readable  and  writable, 
as  well  as  machine -processable.  Such  approaches  allow  the 
machine  to  perform  computational  reasoning  over  a  knowledge 
base  while  expressing  rationale  that  is  human-understandable. 
The  overall  goal  of  this  research  is  to  use  human-machine 
collaboration  to  enhance  human  cognition. 

Our  recent  focus  has  been  on  behavioural  research  to  test 
the  effectiveness  of  the  technology  for  people  using  simulated 
ISR  tasks.  To  this  end,  we  have  designed  a  platform  for 
running  a  series  of  experiments  in  which  human  participants 
work  alone  or  collectively  on  gathering  synthetic  and  natural 
information  either  in  situ  or  online.  Participants  address  the 
tasks  by  interacting  with  a  CNL  agent  through  dialogues  in 
which  they  can  use  both  natural  language  (NL)  and  CNL.  The 
tasks  involve  the  collection  of  locally  observed  information, 
such  as  simulating  activities  humans  would  perform  on  patrol 


or  while  operating  a  remote  sensing  system.  However,  tasks 
are  simplified  to  allow  participation  without  any  specific  ISR 
training  as  well  as  easing  some  aspects  of  the  natural  language 
processing  (NLP)  performed  by  the  agent  (since  NLP  is  not 
the  focus  of  our  research). 

In  the  experiments,  the  CNL  agent  and  supporting  infras¬ 
tructure  are  designed  to  assist  the  participants  in  their  tasks. 
The  precise  details  of  what  assistance  is  provided  in  each 
case  vary  from  experiment  to  experiment;  generally,  tasks 
involve  information  collection,  sharing,  or  adding  context  to 
local  observations  to  make  them  more  meaningful  (i.e.,  turning 
information  into  knowledge).  Dependent  measures  include  the 
time  to  enter  information  or  make  decisions,  and  their  accuracy 
and/or  quality.  Our  platform  is  called  SHERLOCK  (for  Simple 
Human  Experiments  Regarding  Locally  Observed  Collective 
Knowledge).1 

This  paper  is  structured  as  follows;  Section  II  describes 
the  main  elements  of  the  experimental  world  and  participants’ 
tasks.  Section  III  summarises  the  features  of  the  CNL  agent 
that  participants  interact  with  and  the  assistive  capabilities 
developed  in  our  research.  Section  IV  illustrates  the  use  of  the 
SHERLOCK  platform  with  two  example  experiments.  Finally, 
Section  V  concludes  and  points  to  future  work. 

II.  Experimental  World 

The  world  of  SHERLOCK  experiments  consists  of  sets  of: 
persons  of  interest  (POIs),  locations,  objects,  sensors  (human 
and  machine),  and  attributes  of  all  of  these  entities.  Participants 
are  provided  with  a  subset  of  information  about  the  world, 
for  example,  a  map  of  locations,  “mugshots”  of  POIs,  and 
a  list  of  attributes  of  interest.  Their  task  is  to  collect  specific 
information,  for  example,  to  discover  the  location  of  POIs  and 
the  values  for  their  particular  attributes,  or  just  to  collect  as 
much  information  as  possible.  The  world  is  modelled  in  CNL, 
making  it  relatively  straightforward  to  define  sets  of  instances 
for  a  particular  experiment  setup. 

Worlds  have  information  that  is  synthetic,  natural,  or  some 
combination  of  both.  For  example,  human  actors  may  por¬ 
tray  fictional  POIs  inhabiting  real-world  locations,  or  cartoon 
characters  may  inhabit  synthetic  virtual  locations.  Participants 

1  The  name  was  chosen  to  give  participants  a  sense  that  their  tasks  involve 
elements  of  detection  and  deduction. 


Fig.  1.  The  CNL  agent  used  in  a  SHERLOCK  game 

may  be  tasked  to  report  on  natural  real-world  objects  or 
distinctive  objects  placed  in  locations  for  the  purposes  of 
creating  artificial  scenes.  Given  that  our  research  focus  is  not 
on  NLP,  synthetic  elements  are  often  easier  for  participants  to 
describe  to  the  agent  (with  lower  complexity  and  potential  for 
ambiguity).  For  example,  it  is  easier  to  describe  the  location 
of  an  object  in  cases  where  that  location  has  a  label,  such  as  a 
room  number,  than  it  is  to  describe  an  “unlabelled”  location, 
such  as  a  corridor  or  public  space.  Highly  distinctive  objects, 
such  as  pieces  of  fruit  or  sporting  equipment,  are  also  relatively 
easy  to  describe. 

III.  CNL  Agent  Capabilities 

A  full  description  of  the  capabilities  that  are  the  focus  of 
our  research  is  given  in  [2].  The  core  capability  is  a  conver¬ 
sational  agent  that  supports  a  set  of  interactions  defined  in 
a  protocol,  including  interactions  to  verify  machine-generated 
mappings  of  NL  to  CNL  (confirm),  question-answer  interac¬ 
tions  (ask/tell),  imparting  of  information  (tell),  and  obtaining 
rationale  for  some  piece  of  machine-generated  information 
(why).  Figure  1  shows  a  screenshot  from  one  of  the  mobile 
apps  created  for  a  SHERLOCK  experiment.  The  conversation 
between  the  user  (“Alun”)  and  the  agent  (“Sherlock”)  is 
displayed  as  a  conventional  text  “chat”  thread.  The  user’s  input 
is  in  NL,  which  the  agent  maps  to  CNL  via  NLR  The  user  can 
either  confirm  that  the  CNL  is  an  acceptable  interpretation  of 
his  or  her  input  (shown  as  the  “Yes”  message  here)  or  reject 
the  agent’s  interpretation  and  try  again.  The  agent  can  also  ask 
the  user  questions,  as  shown  in  the  bottommost  message. 

IV.  Example  Experiment  Designs 

Design  1:  “Whodunit”.  This  design  features  six  POIs 
played  by  human  actors.  Participants  are  assigned  a  set  of 
54  questions  in  total,  mostly  concerning  (fixed)  attributes  of 
the  six  characters.  Values  for  the  attributes  —  including  a 
character’s  location,  shirt  colour,  preferred  fruit,  and  hobby 
—  are  discoverable  by  visiting  a  set  of  locations  around  a 
university  building.  In  addition,  a  subset  of  the  54  questions 
requires  participants  to  locate  real-world  objects  in  the  vicin¬ 
ity  of  the  given  locations  (e.g.,  works  of  art  and  historical 


artefacts).  Participants  gain  points  for  submitting  answers 
to  questions  and  for  reporting  other  real-world  objects  in 
the  environment  (including  both  “mundane”  items,  such  as 
furniture  and  office  equipment,  and  a  number  of  “anomalous” 
objects,  such  as  balloons  and  toys  that  would  not  normally  be 
found  in  a  university  building).  The  CNL  agent  mediates  the 
collection  of  information  as  a  crowdsourcing  exercise.  Each 
participant  has  access  to  a  shared  dashboard  where  questions 
are  considered  “settled”  once  a  sufficient  number  of  consistent 
answers  has  been  received.  Different  groups  of  participants  can 
be  equipped  with  different  variants  of  the  CNL  agent  (e.g., 
a  version  supporting  only  confirmatory  and  tell  interactions, 
and  a  version  that  also  supports  ask/tell  querying,  allowing 
participants  to  access  other  players’  submitted  information). 
Gradually,  as  information  is  acquired,  participants  can  uncover 
a  “crime”  committed  by  one  of  the  POIs  against  another. 

Design  2:  ISR  asset  assignment.  In  this  design,  POIs  can 
move  between  locations.  Participants  can  only  view  locations 
via  simulated  sensing  assets  (e.g.,  colour  or  B&W  cameras). 
The  participants’  goal  is  to  locate  the  POIs  (and  maintain 
current  location  information)  and  report  on  visible  attributes  of 
the  POIs  (e.g.,  carried  objects,  shirt  colour).  Participants  can 
work  alone  or  in  a  coalition  team  with  sharing  of  assets  and 
information.  The  agent  capabilities  vary  in  terms  of  the  degree 
of  intelligent  and  predictive  assistance  given  to  participants  in 
terms  of  asset  selection,  using  state-of-the-art  algorithms  [3], 

V.  Conclusion 

Initial  results  from  SHERLOCK  experiments,  currently  be¬ 
ing  written  for  publication,  indicate  that  untrained  users  can 
quickly  become  productive  with  the  CNL  agent,  allowing 
them  to  cooperate  effectively  with  other  “coalition  members”. 
So  far,  approximately  45  people  (undergraduate  students  and 
industry  researchers)  have  participated  in  SHERLOCK  exper¬ 
iments.  Further  experiments  will  be  run  through  April  2016. 
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